Allow overriding schedulerName on worker/tasks pods#53983
Merged
jedcunningham merged 1 commit intoapache:mainfrom Jul 31, 2025
Merged
Conversation
SamWheating
requested review from
dstandish,
hussein-awala and
jedcunningham
as code owners
jedcunningham
approved these changes
Jul 31, 2025
ferruzzi
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Aug 7, 2025
fweilun
pushed a commit
to fweilun/airflow
that referenced
this pull request
Aug 11, 2025
98 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We have been using multiple kubernetes schedulers in order to improve the autoscaling performance of our clusters. This includes using separate schedulers for evictable pods (such as the airflow scheduler or webserver) and non-evictable pods (such as spark drivers or Airflow kubernetesExecutor pods).
Currently the Airflow helm chart allows us to set the schedulerName (added in #33843) but this can only be set globally across all airflow components. We would like to override this for only the workers, similar to how other fields such as node selectors and affinity can be overridden.
For consistency with other configuration options under
workers, I have added this override to both the pod template file and the celery workers deployment.We could also support configuring an overriding schedulerName for all other components (webserver, scheduler, etc) but I can't think of any need for this currently.